feat(lakehouse-agent): flag-routed Okta + OBO alongside Cognito - #1933

…naming

ported from validated fork; two-app OBO topology; project-name + SSM-prefix normalized to lakehouse-agent (OBO feature terms preserved); ruff-formatted to repo style (46 F541 cleared); py_compile clean, zero -obo-project survivors.

net-new; search_claim_notes + owner_user_sub per-user RLS; least-priv AOSS role; Okta JWT authorizer + Authorization-header allowlist; normalized to lakehouse-agent; ruff-formatted (27 F541); hygiene-clean; generated .bedrock_agentcore.yaml excluded.

net-new; AOSS collection provisioning + CustomOauth2 OBO provider (TOKEN_EXCHANGE, RFC 8693) + OBO gateway/target (no interceptor by design) + optional agent-IAM (direct self-mint path only; gateway path needs none - Finding 15) + cleanup; normalized to lakehouse-agent; ruff (57 F541); hygiene-clean.

…Phase 1)

DR-8 flag-branch convention; authorizer/config/claim-extraction branched Cognito|Okta (Cognito byte-preserved); RESPONSE interceptor attached (A1, both paths); in-place idempotent role/gateway (Flag-3); IDP_PROVIDER plumbed to interceptor Lambdas via deploy env (Flag-2); GW1 keeps upstream gateway-* keys (Flag-1); fixed latent Okta config-print AttributeError; branched token_exchange/tool_validation group-claim. ruff/py_compile/hygiene clean.

thin notes REQUEST interceptor injects caller sub on body-context; server reads context.user_id (Cognito) vs bearer (Okta), fail-closed both; match-by-construction seed (cognito-user-*-sub); 5b/04 GW2 branch (Cognito interceptor gateway + M2M provider + OpenSearch target / Okta OBO verbatim); OpenSearch-runtime authorizer branch; dedicated least-priv notes-interceptor role. ruff/py_compile/hygiene clean.

two-client wiring (claims/->GW1 gateway-arn, notes/->GW2 obo-gateway-url), same bearer to both, union catalog, finally-teardown; IdP-agnostic (GW2 keys unified by 5b/04); agent role net-zero OBO (Finding 15); fail-soft claims-only if GW2 absent (now logged).

…n paths

Paired with 4b/5b; all cross-refs fixed (notebooks, both READMEs, ported dangling refs now valid); advanced-interceptor add-on hard-coded paths patched (pre-deploy.sh + policy-stack.ts + README) — paths only, no logic/Cedar; repo-wide grep for old names = zero; add-on pre-deploy.sh bash -n + path-existence verified.

obo-gateway-* SSM keys → notes-gateway-*; resource → lakehouse-notes-gateway; derived role → agentcore-lakehouse-notes-gateway-role; writer(04)/cleanup(06)/reader(agent); Okta-only obo tokens preserved (lakehouse-obo-target, lakehouse-obo-okta-provider, okta-obo-client-*, obo-credential-provider-arn); 05_update_agent_iam uncoupled (patches agent role, not gateway); grep obo-gateway=zero except the kept 5b-obo-gateway-setup dir; py_compile OK; ruff format --check clean.

… chunk 3)

detect_gateway_idp (discoveryUrl-host primary + credential-shape fallback, fail-closed) + assert_gateway_idp_matches; guard inserted before the converge/reuse in create_gateway.py (GW1) and 5b/04 (GW2, both branches) — where the old code would silently keep the wrong-IdP authorizer; fail-fast 'run teardown first' message; 10/10 offline unit test. DR-11. (Pre-existing ruff nit at idp_config.py:65 is NOT ours — verified append-only diff; consistent with repo baseline.)

…chunk 4)

Rename 01-deploy-cognito→01-deploy-idp; Step-0 set+persist IDP_PROVIDER (set_idp_provider, default cognito, R1.4); ## [COGNITO]/## [OKTA] headers + if-guards on setup/audit/verify (R11.2/.4); Cognito cells byte-preserved (DR-8); Okta verify reuses verify_okta_setup.py; dropped unused cognito_client from agnostic init; stale-ref + rename cross-refs fixed, grep-zero; 18 cells valid nbformat, code compiles. Satisfies R1/R2/R9.1/R9.2/R11.

…unk 5a)

git-rename 05-deploy-gateway->05a-deploy-claims-gateway; retitle GW1/claims (both paths); removed stale DynamoDB-tenant-map prose (cells 1+13) -> REQUEST+RESPONSE interceptor design; agnostic-ified Step-3 ARNs cell (dropped hard-coded Cognito pool-ARN, reads interceptor+runtime ARNs only); narrative ## [COGNITO]/## [OKTA] notes only (no functional guards -- branching stays in create_gateway.py); cross-refs fixed (04/06/README), grep-zero; 14 cells nbformat-valid, code compiles.

…rch (Phase 6 chunk 5b)

net-new notebook wiring collection->runtime->[COGNITO]seed->load-sample->verify->auth-layer([OKTA]OBO-provider / [COGNITO]interceptor)->GW2 (04, branches internally, DR-11 guard fires there); flag read once via get_idp_provider; 05_update_agent_iam omitted per Finding 15 (note-only cell); seed [COGNITO]-only; seed ordered BEFORE load-sample -- the loader tags sample notes with owner subs and aborts if absent (corrects initial design ordering); 22 cells nbformat-valid, code compiles, cwd paths verified, hygiene clean.

…FINDING-1)

Okta notes path had no producer for okta-user-*-sub before 05b load_sample (the fork's inline notebook-07 fixture was left behind in consolidation) -> Okta 05b Step 4 sys.exit(1). Fix: setup_okta.create_test_users now writes /app/lakehouse-agent/okta-user-<label>-sub = u["login"] (EMAIL = access-token sub claim) for every test user, with a fail-fast @-guard; created.id deliberately NOT used (would vacuous-pass, fork Finding 16). Reconciled stale "notebook 07" refs (loader docstring/error/module-doc, design section 6) -> setup_okta/notebook 01. Cognito seed unchanged (asymmetry justified).

…k-5 gate)

FINDING-2 (MAJOR) forward-trail 05a->05b->06 + 06 lists 05b as prereq (GW2 no longer orphaned); FINDING-3 (MINOR, Option a) IdP toggle stays the in-notebook Step-0 cell (explicit value=), removed the misleading .env IDP_PROVIDER advertisement (.env.example + idp_config docstrings) that created a silent-override trap -- DR-12 UPHELD+clarified; FINDING-4 (NIT) loader comment opensearch/->notes/; plus ruff format idp_config.py (our net-new file, now formatter-clean). R1.4 preserved.

…h-PKCE) + IdP badge (Phase 6 chunk 6a-i)

Flag read once (get_idp_provider), fail-soft banner→nb01 if unset; load_config
branches cognito_*/okta_* + shared runtime_arn/gateway-url(GW1)/region → single
st.session_state.idp_config (replaces cognito_config/auth_config); login
branches — [COGNITO] upstream password-grant BYTE-PRESERVED, [OKTA] ported
Auth-Code+PKCE (OAuth2Component, REDIRECT_URI+coupling note); shared contract
access_token/id_token/user_email; IdP badge; enrichments ported SHARED
(invoke_agent (10,300), fetch_interceptor_tools soft-fail via GW1 gateway-url,
nest_asyncio, PATH_LABELS, last_tools_used, env-gated save-token); requirements
+= streamlit-oauth/PyJWT/mcp/nest_asyncio + boto3 bump; dropped unused Cognito
client-secret from config. Ruff: format-clean; check=21 baseline-class (no new
class) — left per convention.

…08-streamlit + README reconcile (Phase 6 chunk 6a-ii)

Atomic renumber (git mv cleanup→09 first to avoid an 08 collision, then
streamlit→08); 08-streamlit-ui [COGNITO]/[OKTA] guards (Cognito M2M pre-flight
token + invoke test guarded, Okta→launch app, launch cell agnostic, branch-aware
summary), flag read once (c2 get_idp_provider); README reconciled to the final
arc 07-iso-test / 08-streamlit / 09-cleanup (guide-range, notebook-table,
cleanup §, structure-tree, 06 next-links) + added the 07-iso-test row (file
lands in 6c); resolves audit FINDING-5 (cleanup is now 09). grep old names = 0,
no double-08, nbformat valid + code compiles.

…der (Phase 6 chunk 6b-i)

NEW interceptor-notes/cleanup.sh — idempotent teardown of the notes-interceptor (Lambda lakehouse-notes-interceptor, role lakehouse-notes-interceptor-role + policies, log group, SSM notes-interceptor-lambda-arn, local dist/+zip), mirrors deploy.sh, --keep-ssm flag, Cognito-only (all skip no-ops on Okta).

06_cleanup_obo_gateway.py now explicitly deletes the Cognito M2M provider lakehouse-notes-cognito-oauth-provider (delete_oauth_provider generalized to a name param, called for both the Okta OBO + Cognito M2M providers, safe-if-absent both ways) — removes reliance on cleanup_gateway.py's incidental substring match.

Fills the two teardown gaps found by the 6b recon. Verified bash -n / py_compile / ruff (18 baseline-class, no new).

Restructured to reverse-deploy order + dual-IdP; flag read once (c1). NEW Step 2 GW2 teardown (06_cleanup_obo_gateway both-paths + [COGNITO] notes-interceptor cleanup.sh); NEW 4b OpenSearch-runtime cleanup (Step 4); Step 7 IdP branch ([COGNITO] cleanup_cognito / [OKTA] cleanup_okta — guarded; cleanup_okta never unconditional, safe against its sys.exit-without-OKTA-env); DELETE_S3_BUCKET flipped True->False; title/summary refreshed to 01-09 arc + full coverage.

B17 PRESERVED — no put_data_lake_settings/LF-admin removal (deregister only; verified 0 in code cells). Closes audit Note-1. Verified nbformat(25)/compile(14)/all-cwds-exist/AST-guards.

… 6 chunk 6c)

Ported the fork's isolation test (cross-user data isolation on both gateways). FINDING-1 closed (fixture READS <idp>-user-*-sub; put_parameter=0). Cell-19 token-validation branched (Okta aud+email / Cognito GUID no-aud). Cell-25 log-scan advisory fail-soft on Cognito. SSM prefix normalized + keys renamed (notes-gateway-url / gateway-url). Cleared all outputs (DR-6); dropped empty cell (42 cells). requirements.txt += opensearch-py/aws-requests-auth/PyJWT/nest_asyncio.

Cells 28-30 (tool-gating) [OKTA]-guarded pending TRM-GAP; core scenarios 1-6 IdP-agnostic + independent. Verified nbformat/compile/grep.

…-gating (MAJOR)

(a) setup_dynamodb_tenant_role_maps.py branches claim_name on IDP_PROVIDER (cognito:groups/groups; read once via get_idp_provider) — the Okta interceptor lookup now hits (was Cognito-only → Okta claims tool-gating returned empty/403); claim_value (IdP-invariant) + allowed_tools (admin=query_login_audit/text_to_sql, correct for FINAL) unchanged.

(b) 07 cells 29-30: fork-only get_claims_analytics -> query_login_audit + promoted cells 28-30 to dual-IdP (removed [OKTA] guard; legit cell-6/cell-19 branches remain). get_claims_analytics NOT re-added (DR-2). Live group-claim delivery = Phase-7 validation.

Closes MAJOR finding TRM-GAP. Verified py_compile/ruff(14 baseline, 0 new)/nbformat/grep.

Flag-map 'How to set it' -> DR-12 Step-0-cell mechanism (dropped the .env contradiction); 06 row claims/+opensearch/ -> claims/+notes/; dropped stale build-progress meta; 09-cleanup row -> dual-IdP branched; notebook-table + structure-tree add 05b-deploy-notes-gateway + missing dirs (1-okta-setup/4b/5b/interceptor-notes); overview + streamlit prose -> dual-IdP; test_cognito_login dir-prefix fix.

05a DynamoDB-prose CORRECTED (claims interceptor DOES use lakehouse_tenant_role_map; the earlier 'no DynamoDB' note was wrong; DR-9 thin-notes counterpart preserved). Closes tracked doc items (05b-listing, opensearch/->notes/, 05a-prose). Verified nbformat/grep.

… (Phase 6 chunk 7b-i)

Added 'python -m utils.idp_config <cognito|okta>' entrypoint (argparse choices, explicit value = DR-12-consistent, boto3 imported after arg-parse so --help works offline; library API unchanged). Rewrote the Option-B DEPLOY walkthrough dual-IdP (Step-0 flag-select + [COGNITO]/[OKTA] Step-1 + net-new Step-5b GW2/OpenSearch mirroring the notebook arc). Rewrote the CLEANUP block dual-IdP reverse order (mirrors notebook 09). DANGLING-REF test_cognito_login.py -> decode_user_token.py.

Verified py_compile/--help/ruff(0 new)/grep/fences-balanced.

… chunk 7b-ii)

Rewrote the Cognito-only deployment guide to dual-IdP (508→813 lines):
NEW Step 0 (python -m utils.idp_config flag-select, DR-12); Step 1 branched
[COGNITO] setup_cognito / [OKTA] setup_okta (okta-* keys, seeds
okta-user-*-sub=email); NEW Step 7 Notes Gateway GW2 + OpenSearch (7.1-7.6
mirroring notebook 05b); renumbered 0-10 + all cross-refs; interceptor
sub-steps relabeled 5.1/5.2 (avoid notebook-05b collision); Architecture
dual-IdP subsection (DR-1/DR-9); [OKTA] prereqs; B17 LF-admin union-append
note (§3a) + teardown-preserve note; GW1 SSM keys corrected to
{id,arn,url,name} (verified vs create_gateway.py); dual-IdP reverse-order
Cleanup (mirrors notebook 09; LF admins preserved); Quick Reference +
Directory tree regenerated; fixed stale L63 path.
Verified fences / step-consistency 0-10 / B17-grep / 9 cleanup paths on disk.

… (Phase 6 chunk 7b-iii)

Added a [both IdPs] intro (RBAC is IdP-agnostic; only the group-claim source
differs — cognito:groups/groups, seeder branches on IDP_PROVIDER). Naming
reconcile: lakehouse-role-mappings→lakehouse_tenant_role_map,
lakehouse-session-logs→lakehouse_user_login_audit, interceptor filename fixed.
Mechanism accuracy: rewrote 'How the Interceptor Works' to the real group→role
STS + RESPONSE tool-filter model (replaced X-User-Identity/X-User-Scopes header
framing in prose; box-art diagrams left intact + a one-line 'simplified' caption
under each). Tool accuracy: query_assigned_claims (nonexistent) → real adjuster
set; added text_to_sql to the admin mapping row + Summary (admin =
query_login_audit + text_to_sql; get_claims_analytics correctly absent).
IdP-tagged identity-priority + group claim. Added a concise notes-RLS callout
(OpenSearch owner_user_sub, DR-9) cross-referencing README Step 7 + notebook 05b.
Verified stale-tokens=0 / tool-names in FINAL set / X-User-* only in box-art /
fences even / diagnostics none.

…nt (Phase 6 chunk 8a)

Added Application+Purpose tags at creation on net-new notes-side resources,
correct shape per API: GW2 gateway (both branches, map) + role (list) + Cognito
M2M provider (map); OBO provider (map); 4b runtime role (list) + runtime
(post-launch tag_resource, fail-soft — the starter toolkit exposes no tags=
kwarg as-written); AOSS collection (lowercase key/value; the 3 security/access
policies are not taggable — API gap); notes-interceptor Lambda + role (CLI
--tags). First installment of tag-all (DR-13); claims-side + shared infra
follow in 8b. Verified py_compile / bash -n / ruff format --check / grep.

… — completes tag-all (Phase 6 chunk 8b)

Tagged all remaining deployed AWS resources with Application+Purpose (completes
DR-13 tag-all across 8a+8b): claims gateway/role/provider, shared interceptor
role, request/response Lambdas, 4a runtime role + runtime (post-launch
tag_resource, fail-soft), Cognito user pool (UserPoolTags), post-auth
Lambda/role + login-audit DynamoDB, agent role + runtime (post-launch
tag_resource), S3 Tables table-bucket + 2 tables + query-results bucket
(fail-soft; needs s3tables:TagResource — Phase-7 perm check). Normalized the one
stray Project tag (login-audit DynamoDB) -> Application/Purpose per DR-13.
Gaps (no tags API): Cognito app-client/domain, S3 Tables namespace, Glue LF
table, AOSS policies. DR-8: 9 upstream/shared files touched, ADDITIVE tags only.
Verified py_compile / bash -n / ruff format --check / grep (Project=0).

…audit B1+B2)

Replace the positional 12-template/4-slice pool (crashed at slice [12:15] for the 5th seeded user -> notes path couldn't deploy on the standard 5-user seed) with a canonical NOTES_BY_LABEL map (5 labels x 3 = 15 notes); assign by LABEL not sort-position (B2); add assert_label_coverage fail-fast pre-check (prevents B1 recurring if the seed changes). Notebook 07 prose corrected to label-keyed; count+disjoint answer key unchanged. Verified py_compile / ruff format --check / nbformat-structural + 22 cells compile / logic-check (5 labels, 15 distinct claim_ids, coverage pass + fail-fast). Real bulk_load = Phase-7 live check. Resolves audit B1+B2.

The agent runtime deployer read only Cognito keys -> omitted authorizer_configuration on the Okta path -> toolkit defaulted to IAM -> Okta bearer token couldn't authenticate. Now reads get_idp_provider once; Cognito branch keeps allowedClients=[cognito-app-client-id] (USER app client, NOT M2M); NEW Okta branch builds customJWTAuthorizer{allowedAudience:[okta-resource-server-audience], discoveryUrl:okta-discovery-url}; request_header_configuration hoisted to unconditional (both paths); IdP-aware status prints; pure-IAM fallback only for genuinely-unconfigured. Verified py_compile + AST structural (2 authorizer branches, header depth-0 unconditional, allowedClients x1 / allowedAudience x1, m2m-count=0) + grep. ruff deferred to pre-PR sweep (carry item 7 - no ruff in reachable env). Resolves audit B3.

…terceptor (R3 · audit M1)

RESPONSE interceptor returned the FULL catalog on missing/invalid token, missing claim, or exception (fail-open, inherited from upstream). Now returns an empty catalog (deny-all) on every failure path; the exception handler rebuilds a minimal tools/list response with result.tools=[] from event (not partial locals). Reuses the existing no-mapping empty-list idiom. Dual-IdP branches preserved. Deliberate deviation from upstream fail-open (DR-14) - disclosed in PR notes. Verified py_compile/ruff-format/grep (venv).

The previous note stated the rationale on authority. Cite it instead, so a
reader can check it rather than take our word for it.

A data filter's row expression is a subset of the PartiQL WHERE grammar that
compares columns with constants, permits no PartiQL functions, and is fixed
at filter-creation time -- so no caller-derived value can appear in it. That
is why per-user row scope would need one filter per user, and why the bound
identity predicate is the mechanism instead.

Links the PartiQL support reference and the row-level access control
tutorial, which scopes a reviews table by marketplace with one filter per
value granted to a separate IAM principal.

Also names the mechanism explicitly: both approaches get called 'user-level
permissions', so the old wording read as if a workable feature had been
skipped.

The row-scope claim was restated in eleven places across nine files. Last
cycle corrected exactly one of them, so the sample shipped a corrected note
next to ten stale ones -- and the next correction would have needed applying
eleven times again.

Canonical now: setup_lakeformation_permissions.py holds the full reasoning and
the AWS references for code readers, and README.md gains a reader-facing
section for people who never open the deployment scripts. Every other site
carries one short consistent sentence pointing at the script.

Applies the accurate mechanism throughout: a data filter's row expression
takes constants only and permits no PartiQL functions, so it cannot reference
the caller -- per-user scope would need one filter per user. Drops the
'documented tutorial limitation' and 'machinery exists but is uninvoked'
framing, which invited the reading that row isolation is untested.

README also notes that both approaches get called 'user-level permissions',
for readers arriving expecting Lake Formation row filters.

Citations follow existing repo convention: 'Reference:' blocks in docstrings,
inline links in markdown.

…down

Fills the three documentation gaps in the Okta path (worklist item 23) and
discloses one code-level hazard found while doing it.

API-token permission scope -- the one real gap. Every doc named the token and
where to create it; none said what it must be able to do. An Okta API token
inherits the role of the admin who creates it, and setup_okta.py writes to four
admin surfaces: authorization servers, applications, groups, users. There is no
pre-flight permission check, so a narrower token provisions part of the tenant
and then fails with 403, leaving a half-built setup. Now stated as a table of
the four surfaces, with Super Admin (or an equivalent custom role) as the
instruction and verify_okta_setup.py as the recovery path.

Plan-name currency -- "Okta Integrator Free Plan", matching the sibling
AWS-operations-agent/okta-auth reference and Okta's current branding, with the
former "Developer Edition" name in parentheses so either search term lands.

What teardown removes, and the one thing it cannot. Cleanup is a full reset on
both IdP paths: 09-optional-cleanup Step 8 runs cleanup_okta.py, which deletes
both apps, the authorization server, three groups, five test users and twelve
okta-* SSM keys. That was undocumented, so a reader had no way to know the
teardown reaches into their Okta tenant, and no way to know that skipping that
one step is how you keep it. Documented in three places, with the direction of
the surprise reversed at the end: the reader's OKTA_API_TOKEN is NOT removed and
cannot be -- they minted it by hand, so the sample never owned it and deleting
the okta-api-token SSM key removes a copy, not the credential. Revoking it is
now an explicit final step in the notebook's summary rather than an omission.

Idempotency is adopt-by-name, and teardown deletes what it adopted. setup_okta.py
reuses any object whose label matches and prints an informational "already
exists" line; cleanup_okta.py deletes by those same constants, and no provenance
is recorded anywhere durable, so the two cannot be told apart at teardown. The
app and server names are distinctive and @example.com is reserved, but
administrators / adjusters / policyholders are ordinary group names in a sample
aimed at insurance teams. Disclosed in all three places with the mitigation
(rename, or use a dedicated free tenant); the code fix is deliberately left to
its own change since it alters the idempotency contract.

.env destination made explicit. .env.example ships at the sample root; the
loader also accepts deployment/.env and prints which file it read. Verified
against utils/env_file.py find_env_file() rather than assumed -- both layouts
are supported by design, so the docs now name the sample root without implying
the other is wrong.

Files: README.md, deployment/README.md, deployment/1-okta-setup/README.md,
09-optional-cleanup.ipynb (3 cells: intro, Step 8, final summary), .env.example.
No behavioural change -- documentation and notebook prose only.

Findings from the first sweep of the shipped reader-facing docs against the
implementation (31 behavioural assertions examined: 20 confirmed with
file-and-line citations, 6 contradicted across 14 sites, 5 unverifiable).
Fixed in severity order. No code changes -- in every case the code was right
and the prose had gone stale.

1. The RESPONSE interceptor was described as doing "row/field filtering ...
   + result scoping" (05a cells 1 and 13). It does neither. lambda_function.py
   returns early on `if method != "tools/list"` and the only field it ever
   rewrites is result.tools. Corrected to match cell 5, which was already right
   three cells away, and cell 5 is now the canonical statement: it says
   explicitly that nothing else is touched, names where row and column scope
   actually live (the caller-bound WHERE predicate and the Lake Formation grant
   on the assumed tenant role), and records the fail-closed empty-catalog
   behaviour. This one mattered most: a reader could have concluded row scope is
   enforced twice when it rests entirely on the bound predicate.

2. Cleanup claimed to "revert the agent IAM OBO patch" (09, three sites -- two
   markdown, one summary print). There is no such step in FINAL and there never
   should be: the agent deliberately holds no OBO grant, so there is nothing to
   undo. 06_cleanup_obo_gateway.py's docstring already said so. Claim removed
   rather than replaced, with a short note pointing at 05b's canonical
   statement so the absence reads as deliberate instead of missing.

3. The adjuster row predicate was documented as assignment only --
   `WHERE adjuster_user_id = '{...}'` (scenarios.md). The code is a disjunction:
   the caller's own claims OR, guarded by a role_exp CTE, the claims assigned to
   them. The code is correct and the doc was wrong: the first disjunct is the
   same row-scope predicate every caller gets, and dropping it would stop an
   adjuster who is also a customer from seeing their own claims. Documented in
   full, including why the branch exists (policyholders are not granted
   adjuster_user_id, so referencing it raises COLUMN_NOT_FOUND) and that the
   disjunct is latent with the shipped fixture, where all nine claims belong to
   the two policyholders. The adjacent "cannot see" list read as exhaustive and
   has been re-scoped.

4. scenarios.md drew column filtering as masking -- five table cells of block
   characters standing in for filtered columns. The code produces absence, not
   redaction: ColumnWildcard.ExcludedColumnNames for adjusters and a 15-name
   column grant for policyholders, so SELECT * cannot return a blanked value.
   Notebook 03 already said "filtering, not masking. Nothing is redacted or
   blanked." The terminology pass that renamed 62 sites never reached the
   pictures, and the picture is what a tester compares their output against.
   Both tables now omit the filtered column and say what its absence means.

   Also fixed in the same block: Scenario 1's "How" line described Lake
   Formation column filtering as "(to be added to Claims table)". It ships and
   is live; the parenthetical said the control did not exist yet.

5. Notebook 07 framed a dual-IdP test as Okta-only in three places, including
   its headline invariant ("any two distinct Okta users ... the same Okta_App").
   It runs on both paths and its own cell 29 says so. Restated in IdP-neutral
   terms with the GW2 mechanism difference called out where it belongs.

6. Notebook 01's Okta prerequisites were the stale copy -- no API-token
   admin-role requirement, superseded plan name. That requirement had landed in
   four other files and missed the notebook that actually runs setup_okta.py,
   which is the closest prerequisite statement to the 403 it prevents. Added,
   with a pointer to the 1-okta-setup deep-dive as the canonical home.

Two prose corrections in the same pass:

- Notebook 02 stated the IAM principal-validation window as "the measured gap
  was four seconds". A second clean-slate deploy needed two backoffs and six
  seconds. Restated as a measured range with the retry budget, and says why the
  2/4/8/16 schedule is necessary rather than generous -- one measurement would
  have justified collapsing it and reintroduced the failure intermittently.

- Notebook 07 asserted "ROPC is blocked on the IE tenant" as though general. It
  is a property of a specific tenant, named in a way a reader cannot check.
  Scoped to per-tenant sign-on policy, with the reason PKCE is used regardless.

Where a claim appeared in several places one is now canonical and the others
point at it: 05a cell 5 for the interceptor's scope, 05b cell 20 for the agent
holding no OBO grant, 03 cell 11 for filtering-not-masking, and
1-okta-setup/README.md for the Okta prerequisites.

Verified: nbformat valid on all five notebooks; cell counts and type sequences
unchanged; every code cell parses except 01's pre-existing `%pip` magic, which
fails identically at HEAD; no ragged markdown tables in scenarios.md; the six
corrected strings return zero matches; hygiene scan clean.

…rming

load_env_credentials() warned on three branches that are all supported
configurations. The first is the one a tester meets on step one:

    WARNING  .env file not found at .env

Contract, established before editing: .env is ONE OPTIONAL RUNG of the
credential chain, not a requirement. init_aws() calls this function and
discards its return value, then calls get_aws_session() unconditionally, which
resolves credentials from container IAM role, then environment variables, then
an SSO profile, then default credentials -- the priority order both this
module's docstring and init_aws()'s own docstring already spell out. A reader
authenticating with `aws configure`, AWS SSO, or an instance/container role
never needs the file, on either IdP path.

So the answer to "is .env needed here?" is: optional, and its absence is fully
covered. That makes the old text misleading rather than merely noisy -- it
reported a non-event as a problem at the first step of the tutorial.

Three branches corrected, all to informational:

- File absent: now says it is optional, names what .env would supply, and names
  the alternatives that work instead.
- File present with no AWS keys: this one bit hardest. A .env holding only
  OKTA_* -- exactly what the Okta path asks the reader to create, and what a
  Cognito reader inherits -- landed here and was told to add four AWS keys it
  does not need. Prescribing an unnecessary fix is worse than a bare warning
  because the reader may act on it.
- File present but setting nothing: same treatment.

Not suppressed. Each branch still reports what it found and what will be used
instead, so someone who INTENDED to supply credentials via .env still gets the
breadcrumb. Wording matches the shape utils/env_file.py already uses for the
same non-event, keeping the two loaders consistent.

Deliberately unchanged: the Optional/Tuple annotations and import order in this
file. ruff reports 12 findings here, byte-identical before and after this change
(9x UP045, 1x I001, 1x UP006, 1x UP035) -- pre-existing and tracked separately;
not folded into a message fix. `ruff format --check` passes, as it did before.

Verified: py_compile clean; ruff check identical to the pre-change baseline;
ruff format --check passes; all three branches exercised live and their output
inspected, including the OKTA_*-only case.

Two items.

1. nb07 asserted nothing about columns (item 20). The isolation scenarios prove
   ROW scope; a Lake Formation column grant that drifted would have sailed
   through a green ISOLATION TEST PASSED, and the 15/20/21 figures existed only
   because they had been checked by hand, twice, out of band.

   Approach: for each tenant role, assume the role, call glue:GetTable on
   `claims` under those credentials, and count the columns that come back. Lake
   Formation filters the table DEFINITION per role, so the returned count is the
   grant -- the same mechanism that makes SELECT * expand to a different width
   per persona.

   Why this shape rather than the obvious alternatives:

   - Counts, not names. Enumerating the expected column list would duplicate the
     grant and leave two copies to keep in step. One number per role detects
     drift in either direction and fails loudly without restating the grant.
     This is the distinction from the SELECT * decision: naming columns in a
     QUERY turns a grant change into a total outage, whereas naming a count in a
     TEST turns it into a loud, specific failure. Different drift modes,
     opposite right answers.

   - Reads the grant rather than a query result, which is what lets it cover all
     three roles. Only policyholder and admin tokens are captured by the
     harness, so a query-based check could not have reached adjusters at all
     without adding another browser login to a run that already has to fit in
     one ~1h Okta token window.

   - No IdP branch, and none possible. Grants attach to the IAM tenant roles,
     which are identical on both paths; the IdP only decides how a caller maps
     onto one of those roles. So the assertion cannot silently cover one path
     only. It also needs no persona tokens, so it still runs when token capture
     has failed -- which is when you most want to know the data layer is intact.

   Failure-mode separation is the load-bearing part, because this ships without
   ever having run live. A count that comes back and disagrees reports
   "policyholders returned 14 columns, expected 15" under a COLUMN GRANT DRIFT
   banner. A count that could not be obtained reports "could not determine
   column count -- <reason>" under an ASSERTION COULD NOT DETERMINE COLUMN
   COUNTS banner, which says in as many words that it is not a drift result.
   Both fail the run -- neither is ever a silent pass, matching the existing
   INCONCLUSIVE discipline -- but they call for opposite next actions, so they
   must be distinguishable without reading code. An absent or empty Glue column
   list is deliberately treated as unknown rather than as a count of zero, so a
   response-shape change cannot masquerade as total revocation.

2. opensearch_tools.py documented search_claim_notes' parameter as the "Okta
   sub" in a function that serves both paths (server.py branches Cognito/Okta).
   Corrected to name both sources. Same defect one layer down from the notebook
   framing fixed in the previous commit; filed as its own item so it was not
   lost, since docstrings were scoped out of that audit.

Verified offline, since the stack is down: nbformat valid; 45 cells, every code
cell parses; ruff check and format clean on the touched module. The new cell's
logic was exercised against fakes across all seven branches -- happy path, drift
in either direction, SSM unavailable, assume-role denied, glue:GetTable error,
and an empty column list -- confirming each produces the right tag, the right
banner, and that the drift and could-not-determine paths are mutually exclusive.

Unproven until the next Okta cycle: the live AWS calls themselves. Specifically
the CatalogId form ({account}:s3tablescatalog/{bucket}, copied from the MCP
server's working call), that the tenant roles are assumable by the notebook's
caller, and that Glue returns LF-filtered columns to an assumed tenant role as
expected. Any of those being wrong surfaces as COULD NOT DETERMINE with a named
reason, not as a spurious drift report -- which is exactly what that separation
is for.

Replaces the third-party OAuth component on the Okta path. That component drives the
flow through a popup and polls the popup's location from inside its own iframe; when
the browser refuses that cross-origin read it discards the security error and keeps
polling, so the sign-in never completes. The behaviour is in the component's own
shipped bundle and is identical across the Streamlit versions tested, so there is no
version to pin.

The app is already a web server, so it can run the flow itself and no popup is
involved. Its own origin is the redirect target, which is the URL the Okta setup
script already registers, so no identity-provider configuration changes and existing
test organisations keep working.

What the flow does now:

- Builds the authorization URL with PKCE S256: a high-entropy verifier is kept
  server-side and only its SHA-256 challenge is sent. `prompt=login` forces a fresh
  credential prompt so a reader can switch test users, and an optional email is passed
  as a login hint to pre-fill the username.
- Receives the authorization code as a query parameter on the app's own page, then
  exchanges it at the token endpoint using HTTP Basic client authentication, as a
  confidential client.

Three details that are load-bearing rather than incidental:

- The verifier and state must outlive a full page navigation, and that navigation
  starts a new session, so session state is gone by the time the code arrives. They
  live in a server-side store keyed by state instead.
- That store is treated as security-relevant. The returned state must resolve to an
  entry we issued, entries are single-use so a replayed code cannot be paired with the
  same verifier twice, and entries expire so an abandoned attempt does not leave a
  verifier resident. The attempt is minted once per session rather than once per
  rerun, so keystrokes in the hint field do not accumulate entries.
- The query parameters are cleared after a successful exchange. Reruns are frequent,
  and a code left in the URL would be resubmitted against a code the provider has
  already consumed.

Scope of the change: confined to the Okta branch, and the session contract both
branches converge on (access token, id token, display identity) is unchanged. The
Cognito password-grant path is untouched. The OAuth component is dropped from the UI
requirements; the flow now needs only the standard library and the HTTP client already
in use.

The sign-in link was rendered with st.link_button, which always opens a new tab. The
flow itself worked, but the tab the reader started in was left showing a logged-out
login form while the signed-in session lived in a different tab. That looks like a
failure even though the sign-in succeeded, and it is a regression against the previous
behaviour, where the sign-in window closed and the original tab became signed in.

Rendered as an anchor with target="_self" instead, so the current tab navigates to the
identity provider and returns signed in. The mechanism is unchanged: the return trip is
still a full page navigation, so the verifier and state still come from the server-side
store rather than from session state.

The hosted-domain step could report success while leaving the user pool with
no domain at all, producing a deployment that completes with every step green
and cannot serve a single request: with no domain there is no /oauth2/token
endpoint, so the machine-to-machine client-credentials grant fails, so both
gateway targets fail to authenticate, and the agent answers nothing. No error
appeared anywhere -- the only signal was an informational line reading as
though idempotency had kicked in.

Three separate causes, each addressed here:

Name derivation. The domain suffix came from a fixed-length slice of the whole
user pool id, which begins with the region, so "us-east-1_UkB4mHbEr" yielded
"useast1u" -- seven characters of region and one of identity. That is a 36-wide
space in a namespace shared by every AWS account in the region, so unrelated
deployments collide readily. The suffix now comes from the pool id's unique
segment, the part after the underscore.

Error handling. The except block treated any error whose text contained the
word "domain" as benign, which covers nearly everything an API whose own
parameter is named Domain can raise. In particular it swallowed "Domain
already associated with another user pool" -- the case where the name is taken
and the pool ends up with no domain. Only a domain that already belongs to
this pool is benign now; anything else fails the step with the underlying
message and a note about what breaks downstream.

Hostname confirmation. The stored hostname was composed from the name the
script intended to use, which asserts success rather than establishing it. It
is now read back from the API after creation, so what reaches SSM is what the
service actually holds. cleanup_cognito.py already reads the domain from the
API before deleting it; this applies the same pattern on the way in.

Verified: the derivation is covered by an offline test that lifts the three
derivation lines out of this file and exercises them against sample pool ids,
including the specific id that collided and two pools that share a first
character. Formatting, lint and compile checks pass.

Running notebook 07 could make the Streamlit walkthrough impossible to
complete, and reported success while doing it.

set_test_user_passwords.py resolved its password as

    os.environ.get("LAKEHOUSE_TEST_PASSWORD") or _generate_password()

so with no environment variable set -- the default, since nothing instructs
the reader to set one -- it generated a random password, applied it
permanently to all five personas, printed "Set permanent password for 5/5
test personas", and deliberately never printed or stored the value. Notebook
07's prep cell calls this helper, so simply running the isolation test
replaced every persona's credential with one the reader could not learn.
Notebook 08 then could not be completed, and nothing said why. Meanwhile the
sign-in panel advertised TempPass123!, which was by then wrong.

Three changes, and the coupling between them is the actual fix:

The fallback is now the documented default. With no environment variable the
helper uses the same value setup_cognito.py already assigns as the initial
TemporaryPassword and that notebook 08 already documents. This restates a
published demo credential rather than introducing one, and it keeps the
default reader's path working with no prerequisite. Refusing outright was
considered and rejected: the helper is called from notebook 07, so refusing
would stop the isolation test running at all for the default reader, which
moves the breakage earlier and wider rather than fixing it. The random-value
path is removed rather than left unused.

The helper now names its source. It prints whether the password came from the
caller, from the environment, or from the documented default -- never the
value. This is the fix for the failure mode rather than for the immediate
bug: the value used and the value documented can legitimately differ, and
when they do, the only way to sign in is to know which one won. That
information previously existed nowhere a reader could reach it.

The sign-in panel no longer displays a password, and the notebook documents
it instead, immediately before the reader is told to sign in -- including the
override rule, so the instruction is true whether or not the environment
variable is set. The app cannot know which value the setup scripts actually
applied, so a credential hint rendered there can silently disagree with
reality; a hint that is confidently wrong is worse than none, because it is
the one value a reader will try.

The three places that state this password -- setup_cognito.py, the helper and
notebook 08 -- must agree, and a comment on the constant says so.

Verified offline, 27 checks, no AWS: resolution order (caller > environment >
documented default) exercised through the shipped function with fake clients;
the generated-password path proven absent rather than merely unreachable; the
source named and the value never printed on all three paths; and the four
files that mention the password asserted to agree, since nothing was
individually wrong before -- the pieces simply disagreed. Formatting, lint,
compile and notebook validation pass.

Also in this commit, because it touches the same panel and serves the same
goal: the active identity provider is now emphasised in the sidebar rather
than captioned. It already rendered from shared code above the login widget,
so both paths showed it, but as de-emphasised grey text. Its job is to answer
"which path am I on?" at a glance -- the two login experiences differ by
design, a password form on one and a redirect to the provider on the other,
and without a legible indicator someone reporting a problem cannot say which
they were looking at. That happened during testing: the panel was described
as looking completely different and it took a source read to establish which
branch had rendered.

The two identity-provider paths should match rather than merely resemble each
other. Both UI changes here follow from that: a credential hint that exists
on one path only is asymmetric by construction, and an indicator that is
present but hard to read does not let a reader tell the paths apart.

setup_cognito.py printed both the user app client secret and the
machine-to-machine app client secret in the clear at the end of a run.

The exposure is durable rather than transient. The deploy notebook runs
this script with subprocess.run(...) and no capture_output, so the
child's stdout is inherited by the Jupyter kernel, captured as that
cell's output, and saved into the .ipynb file. A reader who runs the
notebook and commits their work commits a live OAuth client secret into
their own repository. Console output also gets pasted into issues and
chat threads.

Nothing is lost by removing the values. Both secrets are already written
to SSM Parameter Store as SecureStrings earlier in the same run
(cognito-app-client-secret, cognito-m2m-client-secret), and every
component that consumes them reads them from there. No notebook, README
or script asks the reader to copy the printed value.

Both lines now print a fixed-width mask, matching the convention already
used in create_gateway.py, which prints a mask when it loads a secret
rather than the value. The two CodeQL clear-text-logging suppression
comments went with the lines they existed to silence.

The isolation notebook loaded three persona tokens and asserted nothing
about how long they had left to live or who issued them. The two
policyholder tokens were decoded and checked for distinct subjects and
per-provider claim shape; the admin token was read off disk and never
decoded at all.

An expired token therefore surfaced much later, at the runtime
authorizer, as an empty tool list or a missing row. In an isolation test
an empty result looks exactly like the access controls working
correctly, so a missing prerequisite presented as a passing security
property or as an isolation regression, depending on which cell hit it
first. The column-count assertions need no tokens and would still have
passed, which made the run look half-green.

All three loads now go through one helper that asserts the token
decodes, that exp is present and more than five minutes in the future,
and that iss matches the issuer of the active identity provider. Five
minutes rather than zero because the scenarios take several minutes and
a token dying part-way through is the failure being prevented. An absent
exp claim is treated as unknown and fails, never as acceptable.

The issuer forms are taken from the request interceptor rather than
rebuilt here, so this assertion cannot disagree with the component that
actually validates the token at runtime.

Each token now reports its remaining lifetime and issuer as it loads, so
freshness is visible before anything can fail. Every failure names the
persona, says to sign in again and how, and states that no isolation
control has been exercised yet, so a stale credential cannot be mistaken
for a test result.

…t for RunTimes

…eployment for RunTimes"

This reverts commit 9812327.

Adds an AgentCore CLI (0.30.0) project at lakehouse-agent/agentcore/ that deploys the claims MCP server, the notes MCP server and the agent as CodeZip runtimes from their existing source directories, plus a pyproject.toml per runtime for the CLI packager. agentcore.json is committed with placeholders only; agentcore/deploy_runtimes.py fills the IdP authorizer, environment variables and role ARNs from SSM at deploy time, writes aws-targets.json from the caller's identity, restores the placeholders, and stores the runtime ARNs where the gateway steps read them. Deployer-specific files are gitignored.

…cution roles

Creates and deletes the three AgentCore Runtime execution roles (lakehouse-mcp, opensearch-mcp, lakehouse-agent) in one script, so role management no longer depends on the starter-toolkit deploy and cleanup scripts. Trust policy, permission policies, descriptions and tags are unchanged from those scripts. Create is idempotent (in-place update of the script's own trust and inline policy); delete skips missing roles. --role limits either action to named roles.

Each runtime directory now holds its own CLI project (deployment/<runtime>/agentcore/), so each runtime deploys as its own stack at the notebook step that needs it. codeLocation is the runtime directory itself; runtime source is unchanged. The single sample-level project is removed; the deploy wrapper stays in place for now. .gitignore rules follow the new paths.

The deploy wrapper moves to deployment/agentcore_cli_deploy.py and takes the runtime to deploy (lakehouse-mcp, opensearch-mcp or lakehouse-agent). It checks that runtime's execution role, fails naming the notebook that produces any missing input, injects and restores that project's agentcore.json, and stores the runtime ARN and ID under the SSM names the later steps read. --destroy deletes that project's stack and SSM keys only.

…ooks

Notebooks 04, 05b and 06 now create each runtime's execution role and then deploy that runtime's CLI project with deployment/agentcore_cli_deploy.py. Notebook 09 destroys each runtime's stack and deletes its role only after the stack is gone. The deploy script now finds the npm AgentCore CLI past the starter toolkit's agentcore inside the virtual environment, so the notebooks work from an activated venv.

READMEs: AgentCore CLI prerequisites (Node.js 20+, @aws/agentcore@0.30.0 exact, uv; Docker no longer needed), the two-agentcore-tools clash, per-runtime role creation, aws-targets.json handling, and why gateways and interceptors stay on scripts. Runtime deploy and teardown steps now call setup_runtime_roles.py and agentcore_cli_deploy.py. Notebook 09 intro and summary describe stack + role teardown. Utility scripts point at the new deploy commands.

The three runtimes now deploy through the AgentCore CLI (deployment/agentcore_cli_deploy.py) with execution roles from setup_runtime_roles.py, so the old toolkit scripts are removed: 4a and 4b deploy_runtime.py / cleanup_runtime.py, and 6 deploy_lakehouse_agent.py / cleanup_agent.py. The starter toolkit is dropped from the root, 4a and 4b requirements.txt; the bedrock-agentcore SDK stays.

References to the removed scripts in comments, docstrings, printed next-step hints and the advanced-interceptor cleanup block now name the CLI commands. The 'two tools called agentcore' notes and the wrapper's clash message now describe a previously installed toolkit.

… failures

The Okta token-exchange troubleshooting section now says where to look when an exchange fails (System Log, event type app.oauth2.as.token.grant) and adds the DPoP condition: both Okta apps must have DPoP disabled or the exchange fails with invalid_dpop_proof. It links the AWS post on on-behalf-of token exchange with AgentCore Gateway, whose Common pitfalls section covers the same gates. The top README gains a Further reading section with that post.

…ts and docs

Comments, docstrings, Markdown and notebook prose carried internal design-record identifiers that a reader cannot resolve, since the design document they point to is not part of the sample. The identifiers are removed; the surrounding text is kept, and reworded only where a sentence depended on the identifier (for example, the pre-flight check is now named as the pre-flight IdP-mismatch guard).

No executable code changes: every edited Python file parses to the same AST once docstrings are set aside, and every notebook code cell is unchanged apart from comment lines. Printed strings and API description arguments were left as they are.

Finishes removing internal tracking references that a reader cannot resolve: requirement numbers, finding numbers and task-list pointers in comments, docstrings, Markdown and notebook prose, plus the design-record tags left in four printed or API strings (the notes gateway description, two test banners and the cleanup summary). Surrounding text is kept and reworded only where a sentence depended on the pointer.

Only string literals and comments change: every edited Python file parses to the same AST with string constants masked, and each changed string is one of these removals.

Comments, docstrings, one README sentence and two printed or API strings pointed readers at numbered sections of a design document that is not part of the sample. The pointers are removed and the surrounding text kept, reworded only where a sentence began with the pointer. Only string literals and comments change: every edited Python file parses to the same AST with string constants masked.

… comments

Eight comments in the gateway setup scripts and interceptor Lambdas pointed at a numbered section of a design document that is not part of the sample. The pointers are removed; where a comment named an SSM key or key family, that name is kept. Only comments change: every edited Python file parses to the same AST.